eXplainable Artificial Intelligence

By Apostolos Kakampakos 03400133, Dimitris Karpontinis 03400135

This presentation about XAI and its various methods is for the course Artificial Intelligence and Data Analysis of the Data Science and Machine Learning Master's program in the National Technical University of Athens.

Artificial Intelligence

But what is AI? As a field artificial intelligence probably has its roots in the seminal paper of Alan Turing about thinking machines [1], with it growing ever since. The most popular subfield is none other than machine learning! As a discipline, machine learning is focused in various learning tasks, like Vision and Language Processing. Many techniques have come out over the years, some rule based like decision trees [2] and some data driven like neural networks [3]. A different direction in artificial intelligence is the field of symbolic AI [4]. In this principle various methods like description logic [5] are used to represent knowledge about the world, in a similar manner we humans do as well. As an added bonus, these methods are human readable, therefore can be understood by humans. This comes at a great price, namely someone must write this knowledge in a highly ordered and detailed form. On the other hand Neural Networks don't really need excessive human effort to accurately represent data, but they need great amounts of data to be able to "learn". In recent years we have seen the rise of Deep Learning [6], which requires immense amounts of data (labeled or not). Impressively it reaches in many tasks, human level accuracies. Despite their impressive results in vision and language tasks, they are considered black boxes! This brings up an important issue. How could we know a black box model like that decides in a sensible manner?! This immediately raises safety and trust concerns for many possible implementations of deep learning, like automated driving, detecting anomalies in medical data, producing medicine and so much more!

Source: https://medium.com/nerd-for-tech/ai-vs-machine-learning-vs-deep-learning-60b3d0611fe9

XAI

XAI attempts to tackle the aforementioned trust and security issues of our neural networks with various methods [7]. As it is discussed in that paper, xai is not only important for trust, but also for detecting adversarial perturbations that could change the prediction. All in all, understanding how a "bad" model is misbehaving can save us hours of training in a grid-search like fashion! Furthermore, if the model interpretations give meaningful explanations, we can claim our model "really" understands the underlying causes, instead of just being a function, which happens to be correctly predicting what we give it.

Defining Terms

Explainer: A model or algorithm that receives a trained model and an input and provides an explanation for the output of said trained model to the given input.

Understandability: It denotes the ability of an explainer method to provide human understandable explanations, without the need to express how the model to be explained treats the data.

Comprehensibility: For an ML model, is when it represents its learned knowledge in a human understandable fashion. (For example a Knowledge Base)

Transparency: We need to have our explanations being produced by an explainable in it self method. If not, why would we trust the explanation?

Audience: The ones who receive the explanation.

XAI Goals

The various goals xai algorithms need to tackle are:

Explanation Types

Main part

Having delved into the various intricacies of XAI, we will now present different explanation methods on a pre-trained model, and a dataset.

Dataset

For our dataset we will use images from the coco dataset [8]. This dataset is full of real-world images containing various classes of objects. This dataset is used for segmenting individual object instances. The Microsoft Common Objects in COntext (MS COCO) dataset contains 91 common object categories. In total the dataset has 2,500,000 labeled instances in 328,000 images.

Having loaded the entire coco 2014 dataset (at least its urls and contained objects), let us see a random image from our dataset!

Every image contains its corresponding url (some urls unfortunately are down), label_ids and label_texts which contain the names of the object said image contains! Let us now present the classifier we will use for all xai methods.

For a classifier, we will use one of the pre-trained conv nets trained on the Places Dataset [9]. Those models, which were used in the paper can be downloaded from the github repo: https://github.com/CSAILVision/places365

We selected to use Resnet18 [10] for our predictor.

Prune and Join

As we have too many pictures, we should reduce the dataset size. We will perform this by picking 2 objects, that are not really related and are not in many pictures. Hopefully it will give us variety of objects and relations. We will pick surfboard and pizza ! Let us present the entire process.

Let us again see a random image! It has to either contain surfboard or pizza!!!

As we can expect, since our image pool is now comprised of very specific object, the possible places our classifier predicts, will be considerably fewer!

Simple Knowledge

To represent an ontology we will use the python library owlready2 [11]. First let us see an ontology of a single image!

As we can clearly see, the image above contains 2 serfers, with their surfboars in the sea! A very simple ontology, would be to simply state that this image contains (person, person, surfboard, surfboard). But, this is a very simple ontology, with basically zero relations! For example one could start generating relations using Wordnet [12]. This lexical database is inspired by modern psycholinguistic theories of human lexical memory!

Wordnet organizes nouns, verbs and adjectives to synonym sets, each representing one underlying lexical concept. We used the hypernym relations (with the relation subClassOf). Therefore we could obtain a bigger ontology structure. This is implemented below!

To process this ontology, besides owlready2, there are a few very useful graph database utilities one may use. To aid us in the analysis of all the graphs, we chose Ontotext GraphDB : https://graphdb.ontotext.com/

Source: GraphDB visualization

As we can see in the above visualization, our knowledge has the basic object 327758 (our image), that is connected with various depictedobject objects. Those represent the actual objects we have in our image! These objects themselves are linked with relations "type" to specific types of object (here surfboard, person). Furthermore, we can see surfboard is linked with the relation "subClassOf" to the object board. This is simply a wordnet relation. The above diagram is purposefully incomplete to aid in our visualization. Now we will show the full diagram!

Source: GraphDB visualization

To understand the structure, let us begin with the root node 327758, representing our image. Using the relation hasObject, we can move to depictedobject3. Then we can hop using the relation type to surfboard. Now we can start moving with relation subClassOf to board -> sheet -> artifact -> whole -> object -> Thing. Thing represents the one class that is a hypernym to all objects in wordnet.

Having constructed an ontology for one image, there is a straightforward way to create an ontology for our entire dataset. We will simply construct an ontology on each image, in such a way as to have distinct depictedobject objects, while attributes like surfboard, pizza, etc will be unique! In this manner the image ontologies will be entangled by the similar attributes they have in them!

Below we will implement this, with our coco dataset!

Having constructed the ontology, we can use SPARQL to query it. For example the query:

SELECT (COUNT(?s) AS ?n_triples) WHERE { ?s ?p ?o .}

gives us 125342 as the number of triples in our knowledge.

Another query could be used to ask which pictures in our dataset contain both pizza and surfboard. The query in SPARQL form is as follows:

PREFIX  coco:<http://coco_dataset/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

SELECT DISTINCT ?s WHERE {
    ?s coco:hasObject ?o .
    ?o rdf:type coco:pizza .
    ?s coco:hasObject ?z .
    ?z rdf:type coco:surfboard .
}

The results are only 2 images, with ids: 211725, 458057. Those will be presented below:

Furthermore, we can utilize more complicated queries in order to discover, which images have the most pizzas in our dataset.

PREFIX  coco:<http://coco_dataset/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

SELECT ?m WHERE {
{
SELECT (COUNT(*) as ?r) WHERE {
    ?s coco:hasObject ?o . ?o rdf:type coco:pizza.}
GROUP BY ?s ORDER BY DESC(?r) LIMIT 1}
{
SELECT ?m (COUNT(*) as ?f) WHERE {
    ?m coco:hasObject ?q . ?q rdf:type coco:pizza.}
GROUP BY ?m}
FILTER(?r = ?f)}

The result is only 1 image, with id = 34865. The image is the following:

Intuitively, we can infer that the model seems to classify images that contain pizza(s) as "pizzeria", provided that other kind of objects are not included, or atleast dont have as many instances in the image.

A simple explanation rule for our predictor like "if the image has pizza then pizzeria", although simplistic, seems to have some utility.

We can enforce a rule like that with a simple query of the form:

PREFIX coco:<http://id_327758/>
PREFIX rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>

INSERT {?x img:classifiedAs coco:pizzeria}
WHERE {
    ?x img:hasObject ?y .
    ?y rdf:type coco:pizza .
}

Explainers

Lime

One method of post-hoc explanation is called Lime [13]. Post-hoc refers to the fact that the method treats the predictor as a black box, only needing its output per input. This explanation technique attempts to understand the model by perturbing the input of data samples and understanding how the predictions change. In this way, we can extract patches of pixels in a picture, that are important for the classification of the entire picture! Some patches are considered positive, as in, they help the model to predict the image correctly and some negative, for the opposite reason.

We shall present the lime utilities, found in the github repo of project lime: https://github.com/marcotcr/lime

Source: https://towardsdatascience.com/black-boxes-and-their-intrusion-620aa3c4c56b

As we can see the model correctly predicts this image as "wave" and it's positively influenced by the water as well as the sky. Furthermore, it seems that it is also influenced positively by the surfers! Moreover, we can see that the mask only has 2 distinct values, which means that we don't really have any negatively influencing patches! A patch would be considered negatively influencing the model prediction, if by perturbing the patch, the prediction would be improved not hindered. Curiously a huge chunk of the image is not really considered as positive, despite containing water.

Saliency maps

Another method of explanations is called saliency maps [14]. Those maps are produced by taking the derivatives ,w.r.t the input image pixels, of the maximum prediction probability. In this way, the method can deduce which pixels are important to the classification of the image using the magnitude of the derivatives! The bigger the derivative (in magnitude), the greater the pixel importance. This explanation method considers the model as white-box, in the sense that it can use its internal structure, i.e hidden layer derivatives!

We used code from the github repo: https://github.com/sunnynevarekar/pytorch-saliency-maps/blob/master/Saliency_maps_in_pytorch.ipynb

Saliency maps, in contrast to lime, consider the entire image! Some areas, have small effect (deep blue) and some areas are considered more important (white and red). It would be of interest to express how the image is represented differently for each mask!

A frequently used metric to find the similarity between two vectors is cosine similarity. Namely, the normalized inner product between two vectors!

The above cosine similarity is neither small or big, meaning that some features are considered important by both methods, and other features less so. This is not a suprise, since those methods have a different approach in how they deal with the model, i.e "black model" vs "white model" approaches.

Let us now see the bounding boxes of the image.

Another use of those maps produced by these two methods is to find the importance of the items placed in an image. Every coco image is equipped with a bounding box for each depicted object. We will use those bounding boxes to produce masks, so as to be able to argue about their importance based on the previously defined 2 masks.

The procedure is as follows:

Relatively the most important of our objects is the first object, which is a person. Below we show which person it is.

There are many ways to evaluate the importance of the bounding box ,in explaining how a model classifies an image. Above, we chose a relative importance metric for evaluating which of the objects is the most salient for our model. The problem is that we don't actually have any absolute metric, to establish with certainty whether an object is actually salient or not. One way to do this would be to examine different metrics, or techniques all together, besides the simple cosine similarity!

With this out of the way, one could also note that the surfboard is also relatively salient, atleast for our saliency map.

Tree explainer

A different approach in explaination methods, would be to train an explainer model the way one does also with a classifier model. Obviously, the explainer would have to be itself interpretable so as to be humanly understantable. The model that comes to mind is simply a decision tree!

The process is as follows:

There are a lot of issues in this process. First of all, we have a vision problem here and decision trees are certainly not the best at classifying images. Furthermore, the explanations that they would provide would be at a pixel level, which surely won't be suitable and interpretable. One could salvage this by training not with the images themselves but with some type of features of the images. Here we used the objects contained in the image in a "bag of words" type of manner. For example, if our dataset only had 3 types of objects, namely "surfboard", "person", "pizza" then an image with 2 surfboards and one person would have a "bag of words" type of features of the form [2, 1, 0]. Similarly, if an image had one person and one pizza, then the features of that image would be [0, 1, 1].

Another issue entirely would be how to actually train the decison tree. Generally, one cares about both training and testing accuracies, so the model would not underfit but also not overfit. The problem is here we have a trained to be explained model and also an explainer model. Is there such a thing as overfitting for an explainer model???

One could argue that the same problem of generalization arises here also. If we want our explainer to accurately explain our predictors we should care about the generalization of those explanations. Also a problem native to the trees is that if left unbounded, a tree's decision path could grow immensely. This would ruin the interpretability of the explanations!

Another problem for decision trees is that providing explanations for many prediction labels would be difficult to interpret. Here we have 365 distinct places classes! Although, we have filtered alot of classes indirectly by selecting our images to contain either pizza or surfboard, the problem remains. A way to deal with this, is to simply filter out all the classes that have a small representation in our dataset. This means that we wont provide explanation for those classes but only for the most predicted ones in our dataset.

Below we provide a simple custom implementation of the above ideas:

As we can see most classes have almost no representation in our predicted labels, so producing explanations for those, seems impossible! Therofore, filtering is a necessity here.

Having done the above steps, we can now get our tree explainer

The above tree gives as an approximate explanation of how the resnet seems to classify the filtered data in our dataset. One should note though, that some classes are missing since we limit our selfes to only a decision depth of 3. If some criterion is True one should go left, else right. For example if someone has an image with 1 pizza, then the criterion pizza <= 0.5 is False, so we go right, then the criterion chair <= 1.5 is True so we go left, then finally the criterion bowl <= 1.5 is True, so the image is classified as pizzeria! If we had also 2 or more chairs and 3 or more dining tables, the image would be classified as a coffee_shop instead!

Obviously these decision rules have some semblance of reasonability, but they are flawed as well. This shouldn't be a suprise, since we only picked as features the objects contained in an image.

One could also increase the maximum decision depth...

As we can see more classes are being explained, but since the decision depth is growing so is the difficulty of interpretability. In principle, the tree needs an amount of pruning. Also, we havent really gained that much in accuracy, but it is important more classes exist in the decision tree!

Finally we always need more data!

A possible modification would be to somehow include the wordnet knowledge we had previously so as to increase the amount of features, consequently distinguishing more classes and increase accuracy and explainability.

Having these rules, one could construct new attributes in the dataset ontology, which would be defined from the decision rules of the explainer. For example a decision path would be represented as >=1hasObject.pizza AND <=1hasObject.chair AND <=1hasObject.bowl, where those are simply the decisions we come across as we traverse said decision path. Then we could define that an image, a pizzeria for example, is equal to a subset of (clause1 OR clause2 OR clause3 OR ...), with every clause representing a decision path in the tree that classifies the image as that label, for example pizzeria!

Conclusions

In this presentation we have examined some xai methods using the coco dataset. Moreover we constructed an ontology for our dataset and performed some SPARQL queries to examine it. It would be an interesting addition to our ontology if we could enrich its relations using techniques such as scene graph generation [15]. Furthermore, we could use modern graph parsing methods such as graph neural networks [16] to access the knowledge we have created. Additionally we could in principle use the enriched knowledge of our dataset to produce more complicated rules that could explain a classifier. For example, by not only using the fact that we have a surfboard and a person, but also a relation like surfing to explain the prediction of a wave. In this way we could distinguish between pictures where, for example, a surfboard exists and is not used for surfing but as decoration so it is definitely not a wave!

Finally, all of the above could be done for the entirety of the coco dataset and beyond!

References

[1] TURING A. M., ``I.—COMPUTING MACHINERY AND INTELLIGENCE'', Mind, vol. LIX, number 236, pp. 433-460, 10 1950. online

[2] Quinlan J. R., ``Induction of Decision Trees'', Machine Learning, vol. 1, number , pp. 81--106, 1986.

[3] Rosenblatt F., ``The perceptron: A probabilistic model for information storage and organization in the brain.'', Psychological Review, vol. 65, number 6, pp. 386--408, 1958. online

[4] , ``Computer science as empirical inquiry: symbols and search'', , vol. 19, number 3, pp. 113-126, 1976. online

[5] Franz Baader, Diego Calvanese, Deborah Mcguinness et al., ``The Description Logic Handbook: Theory, Implementation, and Applications'', 01 2007.

[6] Deng Li and Yu Dong, ``Deep Learning: Methods and Applications'', Found. Trends Signal Process., vol. 7, number , pp. 197-387, 2014.

[7] (Barredo Alejandro, Díaz-Rodríguez Natalia, (Del Javier et al., ``Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI'', Information Fusion, vol. 58, number , pp. 82-115, 2020. online

[8] Tsung-Yi Lin, Michael Maire, Serge Belongie et al., ``Microsoft COCO: Common Objects in Context'', 2014. online

[9] Zhou Bolei, Lapedriza {`A}gata, Khosla Aditya et al., ``Places: A 10 Million Image Database for Scene Recognition'', IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 40, number , pp. 1452-1464, 2018.

[10] K. He, X. Zhang, S. Ren et al., ``Deep Residual Learning for Image Recognition'', Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.

[11] Lamy Jean-Baptiste, ``Owlready: Ontology-oriented programming in Python with automatic classification and high level constructs for biomedical ontologies'', Artificial Intelligence in Medicine, vol. 80, number , pp. , 08 2017.

[12] A. George, Beckwith Richard, D. Christiane et al., ``Introduction to WordNet: An On-line Lexical Database'', International Journal of Lexicography, vol. 3, number , pp. 235-244, 1990.

[13] Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin, ``"Why Should I Trust You?": Explaining the Predictions of Any Classifier'', 2016. online

[14] Karen Simonyan, Andrea Vedaldi and Andrew Zisserman, ``Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps'', 2013. online

[15] Guangming Zhu, Liang Zhang, Youliang Jiang et al., ``Scene Graph Generation: A Comprehensive Survey'', 2022. online

[16] Wu Zonghan, Pan Shirui, Chen Fengwen et al., ``A Comprehensive Survey on Graph Neural Networks'', IEEE Transactions on Neural Networks and Learning Systems, vol. 32, number 1, pp. 4--24, jan 2021. online